System for automated screening of security cameras

ABSTRACT

The present invention involves a system for automatically screening closed circuit television (CCTV) cameras for large and small scale security systems, as used for example in parking garages. The system includes six primary software elements, each of which performs a unique function within the operation of the security system to provide intelligent camera selection for operators, resulting in a marked decrease of operator fatigue in a CCTV system. Real-time image analysis of video data is performed wherein a single pass of a video frame produces a terrain map which contains parameters indicating the content of the video. Based on the parameters of the terrain map, the system is able to make decisions about which camera an operator should view based on the presence and activity of vehicles and pedestrians, furthermore, discriminating vehicle traffic from pedestrian traffic. The system is compatible with existing CCTV (closed circuit television) systems and is comprised of modular elements to facilitate integration and upgrades.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a divisional of, and claims the priority of,application Ser. No. 09/773,475, entitled “System for AutomatedScreening of Security Cameras”, filed 1 Feb. 2001, now U.S. Pat. No.6,940,998, which claimed the priority of provisional patent applicationSer. No. 60/180,323, entitled “System for Automated Screening ofSecurity Cameras”, filed 4 Feb. 2000, the contents of which areincorporated herein by reference in their entirety, and claim forpriority thereunder is made accordingly.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to security systems and, moreparticularly, to an advantageous new system involving methods andapparatus for automated screening of security cameras, as in large-scalesecurity CCTV (Closed Circuit Television) systems.

2. Prior Art

Security systems, as used for example in parking garages, provide one ofthe few areas where an owner may feel that it is necessary to employinstalled security technology to its full capacity. When a securitysystem is installed there may be implicit acknowledgment of the need forreliable dependence on the system and its functioning to full capacity.Its presence implies to the public that they are under the protection ofthe system. If then there is an event of loss or injury that might havebeen prevented had the system been functioning properly and to its fullcapacity, the owner may be confronted with a claim difficult to defend.

Although parking garages provide a valuable benefit and highly desirableor necessary service to the public by offering parking facilities forvehicles of members of the public, they may nevertheless present risk tomembers of the visiting public. Property crimes which have beencommitted in parking garages include auto vandalism and auto burglary;crimes against persons which have been committed in parking garagesinclude purse snatching, strong-arm robbery and, occasionally, assaultand abduction. Multi-level pay garages with tollbooths may offer crimedeterrence because of access control and the requirement to pass atollbooth upon exit. But even parking garages so equipped may beincreasingly subject to risk of auto thefts and auto burglaries whenthese garages are located adjacent to quick escape routes such asfreeway on-ramps or major thoroughfares.

CCTV systems can be an effective security tool when installed andoperated properly as part of security systems in such premises whereoperators of parking garages have a duty to avoid crimes or other lossesor injuries which might otherwise occur. Parking garages, in particular,are good candidates for CCTV coverage because persons are more likely tobe alone and vulnerable than in the higher traffic areas. For a CCTVsystem to operate at full capacity, cameras of the system should bemonitored at all times by security personnel.

A CCTV system of multiple video cameras in a parking garageconventionally has no auxiliary system to make intelligent decisionsabout which camera should be viewed on display monitors. But, it issubmitted in accordance with the present disclosure, decisions aboutwhich camera should be watched, and which to ignore could instead bebased on the content of the video, and electronic auxiliary circuitscould be employed to provide intelligent decisions about which camerashould be viewed on one or more selected display monitors. Furthermore,the intelligent system would be compatible with existing CCTV systems.

Although reference is made herein to garages, garages are only oneexample of premises at, in, or in connection with, which such premisessecurity systems are employed to avoid crimes, losses, injuries or otherundesired occurrences. Merely one example of an undesired occurrence(which may also be referred to an incidence) is unauthorized entry, andexamples of unauthorized entry are typified by vehicular and pedestrianmovement in an improper direction or through an unauthorized portal,space, lane or path. All such premises, whether commercial,governmental, institutional or private, in which a security systems orsecurity device or apparatus of the invention could be employed, will bereferred to herein as secured premises.

Small-Scale Security Systems

A small CCTV system may for example have a few cameras and a displaymonitor for each camera. A single security operator can have acontinuous view of all the monitors, so that the sole operator canassess unusual events in a few seconds while watching the monitors, atleast while carefully observing the monitors. Yet, even in a smallsystem, it is difficult or impossible for one such person to watch thesame scene or scenes continuously. After a few minutes of the same view,what may be termed attention fatigue sets in. After hours on duty, themonitors become to the security person just part of the backgroundclutter. Thus, operator concentration and ability to discern undesiredoccurrences, which may otherwise be evident from the monitor displays,is reduced or lost.

Large-Scale Security Systems

In a large CCTV system having hundreds of cameras, the fatigue factor isextreme for security personnel who must observe a correspondingly largenumber of display monitors. Conventional CCTV control systems have beenproposed which have capability to sequence cameras to monitors inrotation. This allows operators to view every camera in the systemperiodically with a reasonable number of monitors.

For example, in a large, sophisticated metropolitan system having about300 CCTV cameras in garages, 13 security personnel might be needed toview every camera monitor once per minute, even when using a knownsequencing system capable of switching four monitors per operator each10 seconds. In such a system, presenting one view per minute on adisplay monitors will not allow operators to detect quickly occurringevents such as purse snatching. In order to operate 13 securitypositions 24 hours per day, adequate staffing requires about 65 personson staff. Even if resultant high costs of such staffing are sustainable,security personnel cannot practically be expected to maintain asatisfactorily high level of attention for adequate incidencediscernment, because such personnel are presented on the displaymonitors with some 11,520 individual scenes to evaluate during each8-hour shift.

Another known method of handling large numbers of CCTV cameras is tocreate a “wall of monitors.” Using, in a CCTV system of approximately300 monitors, each of 19-inch type, stacked from a point beginning 3feet above floor level and extending to 9 feet above floor level, areach of approximately 137 feet of linear wall space would be requiredby the wall of monitors. Or, if arranged peripherally along the walls ofa room, such monitors would completely line a room of dimensions 14 feetby 60 feet. If operators were stationed 20 feet apart along the wall (orwalls), all camera views could be viewed on the display monitors by atleast eight security personnel. However, if such a wall of monitors 137feet in length were to be employed, it is improbable that any crimeevent or other incident would be seen.

FIG. 1 depicts a human figure, being that of a male 6 ft. in height,standing at one end of a row of 76 equipment racks holding 304 monitors,in order to simulate the appearance and relative scale of a so-calledwall of monitors which would result from this large number of CCTVdisplay monitors. Although the human figure is not drawn to scale, theoperating viewing situation or requirements for such a wall of monitorscan easily be visualized, and will readily be realized as beingimpractical for a large quantity of monitors. Smaller display monitorsrequire less space, but security personnel must then view the smallerdisplay monitors from a reduced distance, in order to be able to discerneach scene.

It is postulated that the number of security personnel operators forwatching display monitors of a large CCTV-equipped security system canbe reduced by using known video motion detectors in combination withelectronics for controlling CCTV switching. However, at some level ofactivity in garages of such a large security system using known videomotion detection techniques, cameras without some detectable motion inthe video are omitted from a switching sequence. While detection by avideo motion detector of the movement of even a single car in a cameraview would cause that camera to be included in the sequence, that samecar driven by a person looking for a parking spot may pass severalcameras, causing the view from each in turn to be presented on anoperator's call-up screen.

Adding motion detection to every camera, and custom software to limitcameras in the sequence to those with motion could reduce the requiredstaff watching cameras significantly. Although no precise data is known,it is estimated that operator attention requirements, which may betermed operator load, would decrease by a factor of two if only thecameras with motion were presented to operators of the system.Decreasing operator load by one-half would nevertheless require sixoperators on duty during the day, that is, as one shift, which wouldrequiring a total operator staff of about 30 persons. Even if thesecurity budget will allow for payment of 30 salaries for operatingpersonnel, the monitoring task would drive these operators to extremeattention fatigue within any given shift.

A previously proposed CCTV system intended to be used with airportparking garages was premised on providing video motion detection on eachvideo camera and using software to control electronic selection of onlycameras providing video output with motion so as to be viewed bysecurity operators. As the number of cameras in the proposed system waspostulated to grow, the weakness of simple motion detection could becomeapparent. Commercially available motion detectors for such a system arefound to be unable to distinguish a person from a vehicle. Thus, forexample, every car passing by a camera could trigger a motion detectorof the system. As vehicles would drive down aisles they would passseveral cameras, and this would result in the presentation on displaymonitors of multiple views of the same vehicle. About six operatorswould be required to be on duty during the day, and the repetitivepresentation of views caused by movement of a single vehicle pastmultiple cameras would cause extreme boredom and resulting lack ofattention.

One known method of monitoring a scene is provided in Ross, U.S. Pat.No. 5,880,775, where pixels of individual frames are compared togenerate a difference value, which value when exceeds a predeterminedthreshold activates a VCR (Video Cassette Recorder) for recording.Another method is provided in Winter et al., U.S. Pat. No. 5,875,305,where video data is analyzed to detect a predetermined characteristicbased on features of a target such as size, speed, shape, or chrominancechanges and subsequent video compression storage. Other methods ofmotion detection, fire detection, and other event-based detection withsubsequent system action for security purposes are numerous and wellknown in the field. However, the known art does not fully address theneed for intelligent camera selection based on a plurality of inputs fordecreasing operator load and fatigue. Additionally, the known art doesnot control CCTV switching for operator viewing. Shiota et al., U.S.Pat. No. 4,943,854, provides a multi-video recorder that allowsselection of a signal from a plurality of cameras, however, without anyimage analysis and based primarily on motion detection sensors.Furthermore, the known art detection methods do not employ the uniqueimage analysis techniques of the present invention for intelligentcamera selection, which are more fully described herein below.

Accordingly, a need exists in the art for image analysis techniqueswhich are much more simplified. Simplified image analysis techniqueswill further allow for real-time image analysis and a more robustsecurity camera screening system.

OBJECTS AND SUMMARY OF THE INVENTION

Among the several objects, features and advantages of the invention maybe noted the provision of a novel and advantageous security system usingnovel, highly advantageous methods and apparatus for automated screeningof security cameras described, and specifically such methods andapparatus which:

are more cost effective than any comparable previous CCTV system;

are capable of use in conjunction with large conventional CCTV systemsoperating at full capacity;

achieve marked decrease of operator fatigue in a CCTV system;

improve security in parking areas and garages and other premises havingvehicular and/or pedestrian traffic within the premises;

function as a so-called intelligent electronic system with capability todirect video camera output to one or more video display monitors onlywhen there is something of logical relevance for viewing by an operator;

are effective to cause CCTV monitor views to be presented to theoperator when video camera view content is of sufficient relevance as torequire human level analysis, through use of intelligent electronicselection of views for each of the multiple CCTV display monitors;

provide a solution to the above-referenced foregoing problems ofoperator use of display monitors for monitoring the view from CCTVcameras of a security system;

achieve in a CCTV system a functional operating advantage in thatobservation by operators of display monitors of the system is much lessboring or fatiguing than hitherto characteristic of CCTV systems;

induce an increase in operator attention span and incidence discernment;

achieve a high degree of premises security at relatively low cost; and

achieve in CCTV security systems a high level of reliable dependence onthe system and its functioning to its capacities to an extent nothitherto experienced.

In accordance with one aspect of the present invention, intelligentcamera selection, which is to say, automatic electronically-controlledselection for presentation on a display monitor in accordance with anelectronic logic protocol, is carried out by an integrated securitysystem having a plurality of CCTV cameras covering another plurality ofaccess controlled areas. When there is an event incident or occurrence,for example, a fallen person, the camera viewing the incident isautomatically selected, i.e., its video output is selected to provide acorresponding display, or call-up, of that camera's view on the displaymonitor of an operator. The selection and call-up of the camera view canalso include an audio notification of same. If there is no eventoccurrence to assess, the display monitor is blank. Because suchautomatic camera call-up functions in response to an event occurrence,operator load is dependent on event activity, without regard to thenumber of cameras in the system.

A primary aim, feature and advantage of the present invention is that asecurity system in accordance with the present teachings is capable ofautomatically carrying out decisions about which video camera should bewatched, and which to ignore, based on video content of each suchcamera, as by use of video motion detectors, in combination with otherfeatures of the presently inventive electronic subsystem, constituting aprocessor-controlled selection and control system (“PCS system”), whichserves as a key part of the overall security system, for controllingselection of the CCTV cameras. The PCS system is implemented in order toenable automatic decisions to be made about which camera view should bedisplayed on a display monitor of the CCTV system, and thus watched bysupervisory personnel, and which video camera views are ignored, allbased on processor-implemented interpretation of the content of thevideo available from each of at least a group of video cameras withinthe CCTV system.

Included as a part of the PCS system are novel image analysis techniqueswhich allow the system to make decisions about which camera an operatorshould view based on the presence and activity of vehicles andpedestrians. Events are associated with both vehicles and pedestriansand include, but are not limited to, single pedestrian, multiplepedestrians, fast pedestrian, fallen pedestrian, lurking pedestrian,erratic pedestrian, converging pedestrians, single vehicle, multiplevehicles, fast vehicles, and sudden stop vehicle.

The image analysis techniques are also able to discriminate vehiculartraffic from pedestrian traffic by tracking background images andsegmenting moving targets. Vehicles are distinguished from pedestriansbased on multiple factors, including the characteristic movement ofpedestrians compared with vehicles, i.e. pedestrians move their arms andlegs when moving and vehicles maintain the same shape when moving. Otherfactors include the aspect ratio and smoothness, for example,pedestrians are taller than vehicles and vehicles are smoother thanpedestrians.

The primary image analysis techniques of the present invention are basedon an analysis of a Terrain Map. Generally, a Terrain Map is generatedfrom a single pass of a video frame, resulting in characteristicinformation regarding the content of the video. Terrain Map creates afile with the characteristic information based on each of the 2×2kernels of pixels in an input buffer, which contains six bytes of datadescribing the relationship of each of sixteen pixels in a 4×4 kernelsurrounding the 2×2 kernel.

The informational content of the video generated by the Terrain Map isthe basis for all image analysis techniques of the present invention andresults in the generation of several parameters for further imageanalysis. The parameters include: (1) Average Altitude; (2) Degree ofSlope; (3) Direction of Slope; (4) Horizontal Smoothness; (5) VerticalSmoothness; (6) Jaggyness; (7) Color Degree; and (8) Color Direction.

Average Altitude

The parameter ‘Average Altitude’ calculates an average value of fourpixels in the center 2×2 kernel.

Degree of Slope

The ‘Degree of Slope’ parameter calculates the absolute difference, inpercent, between the highest average value and the lowest average valuecalculated by Average Altitude.

Direction of Slope

The parameter ‘Direction of Slope’ calculates the direction of the slopebased on the highest and lowest average value calculated by AverageAltitude.

Horizontal Smoothness

‘Horizontal Smoothness’ calculates the consistency of change inhorizontal direction from the lowest pixel to the highest.

Vertical Smoothness

Similar to Horizontal Smoothness, ‘Vertical Smoothness’ calculates theconsistency of change in vertical direction from the lowest pixel to thehighest.

Jaggyness

The ‘Jaggyness’ parameter measures the offset in pixels between odd andeven fields for a given target segmented from a frame of video. Theoffset is then used to determine how fast a target is moving and thedirection of movement of the target. Generally, Jaggyness is a measureof the amount of interlace distortion caused by motion between odd andeven fields of the frame of video.

Color Degree

‘Color Degree’ generally measures how far the color is from gray scale.Zero is equivalent to completely white or completely black, and 255 isequivalent to one color completely.

Color Direction

‘Color Direction’ calculates a color space similar to hue based ontwo-dimensional, (B−R and G−R), color analyses. The two-dimensionalanalysis significantly reduces the number of floating point calculationsover that of hue calculations or three-dimensional RGB calculations, andis a factor in achieving real-time calculation. Generally, ColorDirection is a measure of the tint of the color.

An additional image analysis function, namely ‘Maintain Background’segregates background from moving targets by averaging portions offrames that contain no moving targets. The moving target is furtheranalyzed to discriminate vehicular (or other) traffic from pedestriantraffic.

The PCS system is comprised of six primary software components, allbuilt using Microsoft and Intel tools, including a combination of VisualBasic and C++ software programming languages. The six components includethe following:

(1) Analysis Worker(s);

(2) Video Supervisor(s);

(3) Video Worker(s);

(4) Node Manager(s);

(5) Set Rules GUI (Graphical User Interface); and

(6) Arbitrator.

Video input from security cameras is first sent to a Video Worker, whichcaptures frames of video (frame grabber) and has various properties,methods, and events that facilitate communication with the VideoSupervisor. There is one Video Supervisor for each frame grabber. TheAnalysis Workers perform image analysis on the video frames captured bythe Video Worker and subsequently report activity to the VideoSupervisor. Similarly, the Analysis Workers have various properties,methods, and events that facilitate communication with the VideoSupervisor. The Video Supervisor keeps track of when frames areavailable from the Video Worker and when the Analysis Worker is preparedfor another frame, and directs data flow accordingly. The VideoSupervisor then sends data to the Node Manager, which in turnconcentrates the communications from multiple Video Supervisors to theArbitrator, thereby managing and decreasing the overall data flow to theArbitrator.

The Set Rules GUI permits changing the system rules about what video ispresented to which monitor, for example, changing dwell time for sceneswith multiple people or changing the operator console to receive videofrom a group of cameras. The Arbitrator then receives data from NodeManagers about what activities are present in the system, and receivesrules from the Set Rules GUI about what activity should be presented towhich monitor, and correspondingly arbitrates conflicts betweenavailable monitors and pending activity. The system cameras can also becontrolled by the operator with a PTZ (Pan-Tilt-Zoom) control. The PCSsystem also includes quad splitters, which receive analog video from acentral CCTV switch and provide multiple video scenes on one operatorconsole.

The PCS system interfaces with the existing conventional CCTV systemthrough an interface between the Arbitrator and the port server of theCCTV system. Data flow from the Arbitrator to the port server is via aserial link, and data flow from the port server to the Arbitrator is viainterprocess DCOM (Distributed Component Object Model), a protocol thatenables software components to communicate directly over a network.Interprocess data from the PCS system to the port server of the CCTVsystem includes the camera number to next be selected, outputdestination of next camera selection, commands to set up route fromcamera to monitor, and a message string which allows for futureextensions without revising the interface. Interprocess data from theport server of the CCTV system to the PCS system includes the cameranumber that the operator selected for viewing on another monitor, cameranumber that the operator selected for pan, tilt, or zoom (PTZ), and amessage string which allows for future extensions without revising theinterface.

Data flow between the security cameras and the Video Worker, as well asbetween the quad splitters and the user interface is analog video. Dataflow between PCS system components is similarly interprocess DCOM, withthe flow from the Video Worker to the Video Supervisor and the flow fromthe rules database to the Arbitrator being intraprocess COM (COM), asoftware architecture that allows applications to be built from binarysoftware components.

In a known embodiment of the present invention, there exist three NodeManagers, each receiving data from a Video Supervisor, which in turndirects data flow between one Video Worker and four Analysis Workers.There is one Set Rules GUI, and there can exist only one Arbitrator persystem.

Therefore, it will be understood that in accordance with the inventionthere is provided a novel and advantageous security system, which may betermed a composite security system, in that it comprises both PCS andCCTV subsystems functioning synergistically.

It is also within the purview of the invention to provide, as a systemin and to itself, the features of the present processor-controlledselection and control system (“PCS system”), which can be incorporatedinto, and thus used with, existing CCTV systems and thus becomes anauxiliary system within such a CCTV system.

Additional objects, novel features, and advantages of the presentinvention will become more apparent to those skilled in the art and areexemplified with more particularity in the detailed description thatfollows.

BRIEF DESCRIPTION OF THE DRAWINGS

The above mentioned and other features and objects of the invention, andthe manner of attaining them, will become more apparent and theinvention itself will be better understood by reference to the followingdescription of an embodiment of the invention taken in conjunction withthe accompanying drawings, wherein:

FIG. 1 is perspective view of a so-called wall of CCTV display monitorstogether with the representation of a human figure positioned at one endof the “wall,” in accordance with the known art. The drawing is thuslabeled “Known Art.” The human figure is not drawn to scale.

FIG. 2 is a block diagram of a security system in accordance with andembodying the present invention, having CCTV subsystem components andelectronics subsystem features including software-driven components, bywhich video outputs from video cameras of system are automaticallyselectively made available to display monitors of the CCTV system, wherethe camera views may be viewed by security personnel who observe thedisplay monitors, by video selectively supplied to one video displayconsole or more such consoles. Only a typical unit of possible multipleoperator console positions is shown in this block diagram.

FIG. 3 is a view of image areas used for image analysis according to thepresent invention.

FIG. 4 is a view depicting registration marks highlighted in athree-by-three grid according to the present invention.

FIG. 5 is a view of the basic four-by-four kernel with four two-by-twoquadrants and the pixel numbers in each quadrant for making a TerrainMap in accordance with the present invention.

FIG. 6 is a view illustrating the determination of the Direction ofSlope, allowing 120 degrees to fit into four bits, in accordance withthe present invention.

FIG. 7 is a diagram of a three-dimensional color space used for imageanalysis calculations according to the prior art.

FIG. 8 is a diagram of a two-dimensional color space used for imageanalysis calculations according to the present invention.

FIG. 9 is a color map illustrating the two-dimensional color spaceaccording to the present invention.

FIG. 10 is a view of the offset in pixels between the odd and evenfields for a given target already segmented from a video frame accordingto the present invention.

FIG. 11 is a view showing hatched areas used by an image analysisfunction to count pixels according to the present invention.

FIG. 12 is a view showing an image of only the target without thebackground used by image analysis functions according to the presentinvention.

FIG. 13 is a flow chart illustrating the grab and analyzesynchronization between the supervisor and the analysis worker accordingto the present invention.

FIG. 14 is a hardware block diagram according to the present invention.

Corresponding reference characters indicate corresponding partsthroughout the several views. Although the drawings representembodiments of the present invention, the drawings are not necessarilyto scale and certain features may be exaggerated in order to betterillustrate and explain the present invention.

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee.

DETAILED DESCRIPTION OF THE PRESENT INVENTION

Referring to the drawings, and in particular to FIG. 2, softwarecomponents of processor-controlled selection and control system (PCS) 10are shown in boxes in the upper right area, as contained within thebroken dash-and-dot border. Other components in the figure reflect theblock diagram of CCTV subsystem 12 used in connection with electronicsfeatures including the software-driven components in accordance with theinventive system configuration. The software-driven components of theelectronics subsystem cause video outputs from video cameras of the CCTVsubsystem to be automatically and selectively made available to displaymonitors of the CCTV system, where the camera views may be viewed bysecurity personnel who observe the display monitors, by videoselectively supplied to one video display console for an operator, or tomore such consoles.

Existing CCTV System

It will be assumed for purposes of explaining the new system that itincludes, as in the example given above, hundreds of CCTV cameraslocated within a parking garage or series of such garages or a garagecomplex. Each of CCTV garage cameras 14 is connected directly to one ofthree CCTV switches (two distributed CCTV switches 16, and one centralCCTV switch 18). Distributed CCTV switches 16 forward video from CCTVgarage cameras 14 to central CCTV switch 18. Central CCTV switch 18 isconfigured to be controlled by central switch keyboard 20 in accordancewith known techniques, and directs video from CCTV garage cameras 14 tooperator consoles 22. Distributed CCTV switches 16 and central CCTVswitch 18 receive analog video from CCTV garage cameras 14 andsubsequently send analog video to operator consoles 22. Distributedswitches 16 and central switch 18 are Commercial-Off-The-Shelf (COTS)equipment. It will be understood that there may be other such CCTVswitches of the system.

Various possible types of video input can be provided to central CCTVswitch 18. Such input may include, for example, video from distributedCCTV switch 16, other CCTV switches, and video from other CCTV garagecameras 14.

Central CCTV switch 18 is configured to be controlled by central switchkeyboard 20 in accordance with known techniques. Central CCTV switch 18directs video from CCTV garage cameras 14 to operator consoles 22.Operator consoles 22 are comprised of GUI workstations 24 which may beprovided with quad video splitters 26. Quad video splitters 26 aretypical of such splitters which split video images into a 2-by-2 formatpresenting four video scenes on a single display monitor. In the presentillustrative system embodiment, two of operator consoles 22 are equippedwith quad video splitters 26 intended for monitoring garage cameras andselecting camera views to be transferred to the single display monitor.

The analog video output from quad video splitter 26 is showninterconnected with GUI workstation 24 for illustrating the manner inwhich camera views can be made available for the purpose of setting upand/or changing operation of the system.

Processor-Controlled Selection and Control System (PCS)

Six software modules of PCS system 10 are identified in FIG. 2 andinclude: Analysis Workers 30, Video Supervisors 32, Video Workers 34,Node Managers 36, Set Rules GUI (Graphical User Interface) 38, andArbitrator 40. The functions of each of the software modules and theirinteractions are described in the following:

Analysis Workers

Analysis Workers 30 are ActiveX® EXE modules that are responsible forimage analysis. ActiveX® controls are among the many types of componentsthat use COM technologies to provide interoperability with other typesof COM components and services. Analysis Worker 30 analyze the videofrom one camera and report activity to associated Video Supervisor 32.New frames are obtained from shared memory as directed by VideoSupervisor 32. Analysis Workers 30 are VB (Visual Basic) shellsresponsible for communicating with Video Supervisors 32 and making upperlevel decisions about video activity. Low level calls to the imageprocessing functions are performed from a DLL (Dynamic Link Library), alibrary of executable functions or data. All Analysis Workers 30 in PCS10 share the DLL, and all calls to the DLL are made by Analysis Workers30.

Analysis Workers 30 also act as servers to the Video Supervisor 32. Allimage data manipulation is performed in the C++ functions of the DLL.Within the DLL there exist functions that support the image analysismethods of the present invention as described in greater detail below.

Image Analysis Dynamic Link Library (DLL):

All functions that manipulate image data are in a high level DLL thatenables the rapid creation of image analysis programs from Visual Basicwith minimal effort expended on image data. The DLL processes image dataand returns symbolic data to a Visual Basic calling program, namely, anAnalysis Worker executable. In the preferred embodiment of the presentinvention, the DLL functions exist in three source code modules:

1). Utilities Function (.Cpp)—Contains all utility functions such asread from files and allocate/free memory.

2). Image Processing Function (.Cpp)—Contains image processing functionssuch as Maintain Background.

3). Image Analyses Function (.Cpp)—Contains image analysis functionsthat require prior segmentation.

Arrays are employed in the DLL of the present invention for trackingtargets or objects within the video content. One array includes dataregarding the target (Target Data), and another array includes dataregarding the history of the target (Target History). As symbolic datais collected for the targets, the data is stored in the elements of twodimensional arrays of structures. One dimension is for the numbers offrames to track, and the other dimension is for the number of targets ineach frame, up to a global variable maximum. For example, an element“Name [3] [9]” in the Target History array would hold data for the ninthobject of the frame data stored in row three.

Symbolic data required to make a decision about whether the target is acar or a person is stored in the Target Data array. Accordingly, theTarget Data array holds a number of rows, generally represented by aglobal variable, required to make the decision about the nature of thetarget. The preferred embodiment of the present invention utilizes tenrows in the Target Data array.

Similarly, symbolic data required to interpret the behavior of a targetover a period of time is stored in the Target History array. The TargetHistory array keeps track of the target for several seconds and alsoemploys a number of rows represented by a global variable. The preferredembodiment of the present invention utilizes one hundred rows in theTarget History array.

Each of the Target Data and Target History Arrays have the same numberof columns to track the same number of targets in each frame as definedby the global variable for the maximum number of targets. The preferredembodiment of the present invention utilizes sixty four columns to trackthe number of targets.

The first four elements of the Target Data and Target History arrayscontain the same elements, and the Target History array is longer thanthe Target Data array. For example, ten targets tracked in ten frames ofthe Target Data array are the same targets tracked in the ten mostrecent frames of the Target History array. As a result, data in the tenrows of the Target Data array can always be mapped to the ten mostrecent rows in the Target History array.

The first dimension of both the Target Data and Target History arrays isused as a ring, such that a variable for the current data row will pointthe row of array Target Data to be used for the next frame that isanalyzed. The current data row variable is incremented for each frameanalyzed and when the global variable for the maximum number of rows isreached, the current data row variable is set to 1.

Similarly, a variable for the current history row will point the row ofarray Target History to be used for the next frame that is analyzed, andthe current history row variable is incremented for each frame analyzed.When the global variable for the maximum number of history rows isreached, the current history row variable is set to 1.

As targets are counted and labeled in each frame, the elements of theTarget History array are placed in the corresponding element. Forexample, column 9 of the Target History array will hold data about thetarget with all pixels set to 9 by a label target function.

A further image analysis function is that of Registration Marks, whichprovides an indication of camera movement. The Registration Marksfunction scans through a Terrain Map for corners with high degrees ofslope and different directions of slope than corners in adjacent TerrainMap structures.

The following is a more detailed description of further functions in theimage analysis DLL that manipulate image data:

Allocate Array Memory: A function to allocate memory for the Target Dataand Target History arrays for the sizes specified in the globalvariables for the maximum number of targets, Target Data rows, andTarget History rows. Recall that the number of columns is always thesame for both arrays but the number of rows may be different. The numberof columns is determined by a constant in the DLL and is placed in theglobal variable for the maximum number of targets.

Allocate a Buffer: A function to encapsulate all code required toallocate a specified size buffer using a specified buffer type.

Allocate a Buffer for Color Terrain Map: A function to encapsulate thecode required to allocate a color Terrain Map buffer. A raw buffer isallocated as per arguments which map rows and columns.

Allocate a List for Registration Marks: A function to allocate memoryand return a pointer to a two dimensional array for the RegistrationMark function. The type of structure used is determined by a globalvariable for the number of bits per pixel, the number of rows isdetermined by a global variable for the number of marks, and the numberof columns is determined by a global variable for the number of elementsper mark.

Allocate a Buffer for Mono Terrain Map: Similar to the function forallocating a buffer for the Color Terrain Map, a function to encapsulatethe code required to allocate a buffer for a monochrome Terrain map isutilized.

Target Analysis: A general function for target analysis which outputssymbolic data to the elements of the Target History array as specifiedby various arguments. The arguments according to the preferredembodiment of the present invention include, but are not limited to,whether the target: has a head, is tall, has arms, has legs, istraveling with speed, is traveling in a particular direction, haswheels, is a pedestrian or a vehicle, and when the target last moved.

To determine whether the target has a head, the percent fill of the top⅕ of a bounding rectangle is compared to the percent fill of the secondfrom top ⅕ of the bounding rectangle. If the values are the same, thetarget has no head. If the top is less than 25% of the second, then thetarget has a head.

To determine if the target is tall, an aspect ratio is calculated basedon the height and width of the target. If the aspect ratio is 3 times ashigh as wide then the target is tall.

Referring to FIG. 3, a determination as to whether the target has armsinvolves a series of bounding rectangles 48 over the target 49. Thesecond and third rows of five areas (from top to bottom) of a boundingrectangle is compared to the second and third rows of the boundingrectangle from the previous frame of the target. The level of pixelchange from the current frame to the previous frame determines whetherthe target has arms.

Similarly, a determination as to whether the target has legs involves acomparison of the lower ⅖ of the current bounding rectangle with thelower ⅖ of the bounding rectangle from the previous frame of the target.

Speed is determined by measuring velocity in widths per second andheights per second from the data in the target history array.

Direction of the target is determined by simply comparing the change ispixels between the last frame that the target was recognized and thecurrent frame.

A target is classified as a pedestrian or a vehicle based on multiplefactors, including the characteristic movement of pedestrians comparedwith vehicles, i.e. pedestrians move their arms and legs when moving andvehicles maintain the same shape when moving. Other factors include theaspect ratio and smoothness, for example, pedestrians are taller thanvehicles and vehicles are smoother than pedestrians. To determine when atarget has last moved, a threshold value is used to compare the movementof the target against. If the target has moved more than the thresholdsince the last frame, then a global variable for the last movement isset to zero. If the target has not moved then the global variable isincremented.

A further function exists in the preferred embodiment of the presentinvention to compare two targets to get a probability of whether thetargets in different frames are the same object. The arguments specifythe reference and test targets and support a further function thatcompares targets in adjacent frames to track individual targets.Moreover, the arguments can point to targets in the same frame or totargets an indefinite number of frames apart. The argument returns apercent probability of a match wherein a score of 100% corresponds to apixel by pixel exact match.

An additional function that compares mono Terrain Maps is alsoimplemented to perform segmentation as required by the comparison of twoTerrain Maps. Segmentation is required to distinguish moving objectsfrom the background. Arguments which determine the limit on thedifference between the altitudes before a 2×2 kernel is segmented,independent of the likeness of other terrain features, and how muchdifferent other terrain features must be to segment a 2×2 kernel, evenif the background and test altitudes are the same, are also employed.The absolute values of the differences of the individual terrainfeatures are summed and compared to the argument which determines howmuch different the terrain features must be to segment. If five valuesin a test map are sufficiently different from five values in abackground buffer, then the associated pixel in the result buffer is setto 255, indicating that the 2×2 kernel is to be segmented.

Similarly, a function to compare color Terrain Maps is also contemplatedby the present invention. The argument performs segmentation by thecomparison of two Terrain Maps similar to the argument that comparesmono Terrain Maps as described above and further includes a colordirection. At low color degrees, the direction of the color is givenzero weight.

Additional functions are used to compensate for camera shaking. Offsetsin pixels are determined to indicate the number of Terrain Mapstructures that the frames must be offset from each other to realign thebackgrounds.

A function which confirms Registration Marks scans through the TerrainMap looking for corners that were found by the function that locatesRegistration Marks. Generally, the Registration Marks are located on abackground image and confirmed on a test image. If the camera has notmoved, the marks will be in the same place. If some of the marks arecovered by targets in the test image, others will still be visible if asufficient number are generated.

If the camera has moved, the function that confirms registration markswill search for the new location of the corners in a spiral patternoutward from the original condition until the corner is found or amaximum threshold is reached. If one or more corners can be located withthe same offsets, then those offsets are placed in the global variablesfor x and y offsets, and the number of corners found at those offsetsare returned. If none of the corners in the list can be located, thefunction returns zero. The sign of the global variables for x and yoffset apply to the direction the current buffer must be adjusted toalign with the background buffer after the camera shake. If the x and yoffsets are both −3, for example, then the current buffer must beadjusted down and to the left by three pixels for the remainder of theimages to align.

A further array contains a list of Registration Marks and is a twodimensional array of structures, with one row for each registration markand one column for each Terrain Map structure in the mark. Consequently,global variables for the number of marks and the elements per mark areemployed. The number of marks determines the number of registrationmarks to confirm in the Terrain Map and is the square of an integer. Theelements per mark determines the number of adjacent Terrain Mapstructures to define a registration mark. Furthermore, the size of theTerrain Map is determined by global size variables.

Yet another function interrogates features and provides a means for thecalling process to find out if a particular feature is supported beforecalling it. This function is a switch statement where each case is asupported feature. The switch statement is filled out as the program isdeveloped to recognize such feature names such as:

“HasArms”

“HasLegs”

“HasHead”

“IsTall”

“CheckSpeed”

“CheckDirection”

“RemoveGlare”

“RemoveShadow”

“StopShaking”

“CheckSmoothness”

“ClassificationMatching”

“TemplateMatching”

Targets are labeled using a function that scans through an image thathas been converted to binary (highlighted) with objects ON and thebackground OFF. Connected pixels are labeled in a result buffer with allof the connected pixels in the first target set to one and the secondtarget to 2, and similarly up to 255 targets. Targets having less than aminimum size pixel or more than a maximum size pixel, or less than aminimum height or less than a minimum width are erased. The targetlabeling function will eliminate noise but will not connect targets.

Registration Marks are located using a function that scans through theTerrain Map of the argument looking for corners as indicated by highdegrees of slope with different directions of slope in adjacent TerrainMap structures. The number of elements per mark is a square of aninteger and as low as possible to find clear corners. Each mark willconsist of a square area of the map, for example, a 3-by-3 for thenumber of marks argument is equal to nine marks. The threshold for thedegree of slope and difference in direction of slope is determined bytest and hard coded. As shown in FIG. 4, nine Registration Marks 50 arehighlighted in a 3-by-3 grid.

For each Registration Mark 50 found by the location function, the valuesof the corresponding Terrain Map structures are copied to the elementsof the array having a list of Registration Marks, and the associated rowand column of the Terrain Map are included in the Registration Markstructure.

Identification of target matches with another frame is conducted with afunction that controls the looping through the elements of the two rowsof the Target Data array. The function looks for matches with theanother frame which is assumed to be the last frame, however, anotherframe could be any earlier frame. Every target in the newer frame istested for a match with every target in the older frame, using atwo-stage comparison. First, a fast comparison is performed to see ifthe two targets are similar, and if they are, then the function thatcompares targets is called. A score is then generated and compared to anargument for the required score to indicate whether a match has beenfound.

A function which maintains the background is provided that filtersbackground image data from the targets of interest. Generally, thefunction segregates background from moving targets by averaging portionsof frames that contain no moving targets.

As previously set forth, a function to create a mono Terrain Map is alsoprovided. For each 2×2 kernel of pixels in the input buffer, a TerrainMap is filled out with six bytes of data describing the relationships ofthe 16 pixels in a 4×4 kernel surrounding the 2×2 kernel. As shown inFIG. 5, quadrants are numbered like pixels in each quadrant. Thefollowing are elements used in the MakeTerrainMapMono function:

Average Altitude: Average value of the four pixels in the center 2×2kernel.

Degree of Slope: Absolute difference, in percent, between the highestaverage value of the four 2×2 quadrants in the 4×4 kernel and the lowestaverage value quadrant.

Direction of Slope: Direction of the slope between the highest andlowest quadrants used to define the Degree of Slope. Direction of slopeis determined by the rules according to FIG. 6. The values are one thirdof the degrees to allow 120 to fit into four bits where 360 wouldrequire eight bits.Horizontal Smoothness: A measure of how consistently the pixels changein the horizontal direction from the lowest pixel to the highest.Vertical Smoothness: A measure of how consistently the pixels change inthe vertical direction from the lowest pixel to the highest.Jaggyness: A measure of how much interlace distortion has been caused bymotion between the odd and even fields of the frame.

The resulting Terrain Map is stored in a single plane of structures inrow-column order. The structure type is an array for the terrain dataand has one element for each terrain feature. A buffer for the TerrainMap buffer contains SizeX/2* SizeY/2 structure, and the size of thebuffer is SizeX/2* SizeY/2*size of Terrain Data. The first element ofthe Terrain Map buffer will contain data for the first two pixels ineach of the first two rows of the input buffer, which is the first 2×2kernel found. The Terrain Map buffer is raw, and accordingly there is noheader to provide the size so the function assumes that the globalvariables SizeX and SizeY are applicable to the buffers sent.

Since the top, bottom, left, and right border pixels of the image buffercannot be in the center of a kernel, by definition, data from the firstpass on the first row is used for the top two pixels, not the centerpixels. The second pass is one row down from the first pass to put thepixels of interest in the center of the kernel. Subsequent row passesare incremented by two to keep the four pixel kernel of interest in thecenter until the bottom row, where the increment is one, and the lastrow pass are used to get data for the two bottom rows. The input mapbuffer is assumed to be allocated for the required size.

Similarly, a function is provided within the Analysis Workers to make acolor Terrain Map. For each 2×2 kernel of pixels in the input buffer, aTerrain Map is filled out with six bytes of data describing therelationships of each of the three colors for the 16 pixels in a 4×4kernel surrounding the 2×2 kernel. Quadrants and pixels are numbered asin the function that creates a mono Terrain Map. The color map issimilar to three mono maps with identical elements and an additional twoelements for color direction and color degree as described in greaterdetail below. The following are elements used in the function thatcreates a color Terrain Map:

Average Altitude: Average value of the four pixels in the center 2×2kernel.

Degree of Slope: Absolute difference, in percent, between the highestaverage value of the four 2×2 quadrants in the 4×4 kernel and the lowestaverage value quadrant.

Direction of Slope: Direction of the slope between the highest andlowest quadrants used to define the Degree of Slope. Direction of slopeis determined as shown in FIG. 6, where the values are one third of thedegrees to allow 120 to fit into four bits where 360 would require eightbits.Horizontal Smoothness: A measure of how consistently the pixels changein the horizontal direction from the lowest pixel to the highest.Vertical Smoothness: A measure of how consistently the pixels change inthe vertical direction from the lowest pixel to the highest.Jaggyness: A measure of how much interlace distortion (jaggyness) hasbeen caused by motion between the odd and even fields of the frame.Color Degree: A measure of how far the color is from a gray scale. ColorDegree is zero for full white or full black and 255 for any one colorfully.Color Direction: A measure of the tint of the color. In a color mapknown in the art, yellow is zero degrees, and proceeding counterclockwise, red is 45 degrees, magenta is 90 degrees, blue is 180degrees, and green is 270 degrees. The direction is stored internally as0 to 127.Color Space

Prior art image analysis which employs segmentation based on colordifferences requires a measurement where numbers representing differentcolors have a numerical difference that is proportional to the perceiveddifferences between the colors. Raw RGB (red green blue) values cannotbe used for segmentation because there are three numbers for each RGBset and different combinations of Red, Green, and Blue can be mixed tocreate the same color.

RGB values can be compared by plotting both RGB sets in threedimensional space where the three axes are: Red, Green, and Blue. Asshown in FIG. 7, the origin of the cube where all values are zero isfull black, and the corner diagonally opposite where all values are 255is white. The line between the black corner and the white corner is theneutral axis. All gray scales (From 0,0,0 to 255,255,255) lie on theneutral axis.

The distance from the neutral axis is the measurement of colorsaturation. On the neutral axis, R, G, and B are all equal resulting ina gray scale with no color saturation. At the extreme distance from theneutral axis, (255 as shown in FIG. 2), at least one of the RGB set iszero and at least one of the set is 255, resulting a fully saturatedcolor.

Angular displacement from the neutral axis is the measurement of hue.Equal hues are defined as the surface described by the neutral axis andany point on the surface of the cube. Equal hues correspond to theperception of the “same color” under different conditions. The areasnearest the neutral axis are more washed out or pastel, and the areasfarthest from the axis are more vivid. Areas nearest the black end ofthe axis are as perceived under dim lighting, and nearest the white endas perceived under bright lights.

Using this RGB cube for segmentation, RGB sets that have about the sameangular displacement from the neutral axis are about the same color, andRGB sets that are about the same distance from the neutral axis areapproximately the same saturation. Correspondingly, the threedimensional calculations are computationally expensive and produce moreresults than are used for segmentation by hue and saturation.

As opposed to the prior art that calculates a color space in threedimensions, the image analysis techniques of the present invention useonly two dimensions, namely, Green minus Red and Blue minus Red. Eachaxis is scaled from −255 to +255. Since only the differences areplotted, one position in the plot for each balance in the R, G, and Bvalues results. All of the 256 gray scales in the RGB cube are collapsedinto a single point at the 0, 0 origin of the plot. Likewise each linein the RBG cube representing equal hue and saturation is collapsed intoa single point. As a result of plotting (or calculating) only the valuesof interest for segmentation, this new two dimensional color space plotsall of the 16,772,216 RGB combinations in only 195,075 positions.

In the new color space, Color Direction is equivalent to hue and ismeasured by the angular displacement around the origin of the plot.Color Degree is equivalent to saturation and is measured by distancefrom the origin. Note that all of the gray scales from full black tofull white plot in the same position in the color space, the originwhere there is no color information to use in segmentation.

As shown in FIG. 8, two points are plotted with the same color balance,with Blue being halfway between Red and Green. Green minus Red in onecase is 100, in the other case 200. Since both points have the samecolor balance they plot to the same color direction (27 degrees). Sincethe point where Green minus Red is 200 has more differences in the RGBcomponents, it has a higher degree of color (223 compared to 111).

In the example case of G−R=100, and B−R=50, there are 155 brightnesslevels that will plot to the same position in the color space as Greenvaries from 100 to 255. All of these brightness levels have the same hueand saturation. Brightness is handled in the color space simply as(R+G+B)/3.

In the color map shown in FIG. 9, the two example points fall on a linefrom the point of origin to a point on the perimeter about halfwaybetween Cyan and Green. By examination it may be seen that any linebetween the point of origin and any point on the perimeter passesthrough many saturation levels of the same hue. When used for colorsegmentation, the relatively simple 2D calculation yields the sameresult as the computationally more expensive 3D calculations.

A further function is implemented in the preferred embodiment of thepresent invention to measure the offset in pixels between the odd andeven fields for a given target already segmented from a video frame. Abounding rectangle is determined and a target mask is created, whereinthe target mask is the input to this function. An additional functiondetermines whether a jaggy pattern exists. As shown in FIG. 10, thejaggyness is depicted where the offset in pixels is used to determinehow fast a target is moving and the direction of the target, comparingodd to even fields. Two buffers are allocated and freed by the jaggynessfunction, one for the even scan lines and one for the odd scan lines.The two buffers are template matched to the best fit and the requiredoffsets are placed in argument pointers.

Yet a further function of the present invention removes shadow and glareby utilizing the bounding rectangle of the test image that is given bythe argument row and column of the Target Data array. The boundingrectangle is scanned with 5×5 kernels of pixels. If all pixels in thekernel are marked in the segmented buffer as target pixels, they aretested to see if they are shadow or glare as a group of 25. If thekernel is considered to be shadow or glare, all of the 25 pixels in thesegmented image are set to zero. The following is the test for shadow orglare: The difference array of 25 elements (Background−Test) must all beeither positive (shadow) or negative (glare). The difference(Background−Test) kernel must be smoother than the corresponding 25pixels in either the background or the test image. Roughness iscalculated by adding the differences from one pixel to the next. Aftercalculating the roughness number for the Test, Background, anddifference kernels, the difference must have the lowest roughness (mostsmooth) number to be considered as shadow or glare. The boundingrectangle is reset if pixels are removed from the segmented image. Theremove shadow and glare function can be used with either color or monofiles depending on the headers received.

Another function scans targets in labeled frames by row and keepsstatistics for each target for each ⅕ of the height of the target for:

Smoothness: For each pixel scanned in the target, the correspondingpixel in the original image is examined for a change compared to theadjacent pixel. If every pixel in the original image is different fromthe adjacent pixel, the smoothness is 0%. If all pixels in the originalimage are the same value, the smoothness is 100%. A smoothness number iskept for each ⅕ of the height of the target.

Percent Gap: Counts the pixels of the background that are betweensections of the target. A count is kept for each ⅕ of the boundingrectangle from top to bottom, and is used to deduce the presence of legsor wheels. As shown in FIG. 11, Percent Gap counts the number of pixelsin the hatched area.

Percent Fill: Percent of the bounding rectangle that has labeled pixels.

Percent Jaggy: Percent of the target's Terrain Map structures that haveJaggyness above a threshold value.

While scanning each target, an all black buffer is allocated accordingto the size of the bounding rectangle. While scanning, all correspondingpixels are transferred from the original image that are inside the edgeoutline to the target mask. As a result, an image is produced of justthe target without the background as shown in FIG. 12. If the originalimage is color, only the brightness levels (R+B+G/3) are transferred.

Each instance of Analysis Workers 30 are handled by Video Supervisor 32as an object in an array. There is no arbitrary limit to the number ofAnalysis Workers 30 that Video Supervisor 32 can handle. VideoSupervisor 32 must be in the same machine as Analysis Workers 30 becauseall Analysis Workers 30 operate on image data placed in shared memory byVideo Worker 34 that runs in the same process space as Video Supervisor32.

All communications between Video Supervisor 32 and Analysis Workers 30are handled by the properties, methods and events of Analysis Workers30. Additional functions, properties, methods and events of the AnalysisWorkers may be added to the MotionSentry.DLL to further support theimage analysis techniques as set forth above and communications with theVideo Supervisor as set forth in the following.

Video Supervisor

Video Supervisor 32 modules are ActiveX DCOM components that act asservers to the Node Managers. There is one Video Supervisor 32 for eachframe grabber. Video Worker 34 is an OCX control that plugs into VideoSupervisor 32, and will execute in the same process. In one knownembodiment, the OCX controls will be specific for a Meteor II framegrabber card. The Meteor II frame grabber card has four camera inputsmultiplexed to the same digitizer. The PCS system is configured suchthat frame grabber cards can be interchangeable.

Video Worker 34 maintains four current frames in shared memory, one foreach camera. Video Supervisor 32 keeps track of when frames areavailable and when Analysis Workers 30 are ready for another frame, anddirect traffic accordingly. The interface between Analysis Workers 30and Video Supervisor 32 is generic. If/when the Meteor II frame grabbercard is replaced, only the Video Worker 34 control will have to befurther developed. Analysis Workers 30 are handled as an array ofobjects in Video Supervisor 32. There is no arbitrary limit to thenumber of Analysis Workers 30 that one Video Supervisor 32 can handle.

Video Supervisor 32 acts as a server to Node Manager 36. All calls to aframe grabber DLL are made by Video Worker 34 that plugs into VideoSupervisor 32 and runs in the same address space. All calls to handlethe frame grabber and the associated video buffers pass through theframe grabber DLL. As a result, different frame grabber cards can beemployed with changes only in the DLL.

Generally, the frame grabber DLL includes functions which allocatebuffers for video frames, change the active channel, copy the contentsof one buffer to another, allocate and free frame memory, acquireavailable frames, grab the next frame of video, initialize the videocard, and set the initial configuration and associated control settings.

Video Supervisor 32 coordinates the grabbing of frames with the analysisof frames. Each Video Supervisor 32 controls one frame grabber with oneor more used inputs and as many instances of Analysis Worker 30 as thereare used video inputs. The grabbing of frames between inputs must besynchronized because there is only one digitizer. FIG. 13 shows thegrab/analyze synchronization between Video Supervisor 32 and AnalysisWorker 30. The analysis of frames can be operated asynchronously becausedifferent views, with different targets, can take different times toprocess.

When processing is started, Video Supervisor 32 starts a do-loop,grabbing frames and changing channels. Only one thread is available forgrabbing. If multiple frame grabbers are required in a single computer,then multiple instances of Video Supervisor 32 will be started. Eachinstance of Analysis Worker 30 will run in its own thread because eachis a separate process. Communications between Analysis Workers 30 andVideo Supervisor 32 are handled by setting properties in Analysis Worker30 and asynchronous callbacks to Video Supervisor 32. Communicationsbetween grabbing threads and processing are handled by global arrayswhich generally provide when a frame is ready, when a frame is wanted,and when analysis workers 30 are busy.

Each instance of Video Supervisor 32 is handled by Node Manager 36 as anobject in an array. There is no arbitrary limit to the number of VideoSupervisors 32 that Node Manager 36 can handle. Video Supervisor 32 maybe in the same machine as Node Manager 36, but the program structureassumes that it will be network connected and communicate by DCOMstandards.

All communications between Video Supervisor 32 and Node Manager 36 arehandled by the properties, methods and events of a Super Control Classmodule. The properties generally include commands to start workers, stopworkers, start processing, stop processing, and quit. Correspondingmethods of the Super Control Class module add and drop object referencesfrom Node Manager 36 for asynchronous callbacks.

Callbacks made to Video Supervisor 32 are by properties and methods of aWorkers Report Class module. The methods of the Workers Report Classmodule generally include provisions for busy blocks, to verify thatAnalysis Workers 30 remain on line after no activity, and to notifyVideo Supervisor 32 when Analysis Workers 30 are ready for the nextframe to process.

Additional functions, properties, methods and events of Video Supervisor32 may be added to the frame grabber DLL to further support the framegrabbing techniques as set forth above and communications with other PCSsystem components.

Video Worker

Video Worker 34 is an ActiveX control (OCX) that plugs into VideoSupervisor 32. All calls to the C++ functions in the frame grabber DLLare declared and made in Video Worker 34. All communications betweenVideo Supervisor 32 and Video Worker 34 are through a limited set ofhigh level properties, methods, and events of the ActiveX control.Properties of Video Worker 34 generally include provisions to map blocksof memory, initialize the video card, set or return the active channelof the frame grabber card, execute commands, including, but not limitedto:

Clean Up—Performs all clean up operations such as freeing shared memoryand shutting the frame grabber down.

Grab—Starts grab when current frame is finished.

Grab Frame to Share—Grabs a frame and places into shared memory.

Grab and Show—Grabs a frame and shows on a Video Worker form.

Hide Video Form—Hides the Video Worker form.

Show Video Form—Shows the Video Worker form.

Start Video—Initializes the frame grabber, allocates five frames, andset initial conditions.

Node Manager

Node Managers 36 are ActiveX, DCOM components that act as clients toVideo Supervisors 32 and as servers to Arbitrator 40. The main purposeof Node Managers 36 is to concentrate the communications from many VideoSupervisors 32, and decrease the total traffic that Arbitrator 40 has tohandle. There is one Node Manager 36 for each rack of computers withVideo Supervisors 32. Node Managers 36 handle Video Supervisors 32 as anarray of objects. There is no arbitrary limit on the number of VideoSupervisor 32 servers. Node Managers 36 calculate scores for camerasbased on the events viewed by cameras and also on values set by the SetRules GUI.

Set Rules GUI

Set Rules GUIs 38 are ActiveX, DCOM components that allow changing thesystem rules about what video is presented to which monitor. The systemrules are stored in the rules database 41, as depicted in FIG. 2. Forexample, changing the dwell time for scenes with multiple people, orchanging the operator console to receive video from a group of camerasin a parking structure.

Arbitrator

Arbitrator 40 is the client to Node Manager 36. Arbitrator 40 receivesdata from Node Managers 36 about what activities are present in thesystem, and reads the database regarding what activity should bepresented to which monitor. Conflicts between available monitors andpending activity are arbitrated based on the priority rules, and camerasare called up based on the console to group assignment rules.

Additional System Components

Referring to FIG. 14, additional hardware beyond the standard CCTVsystems includes a video activity processor CPU with a frame grabber foreach four cameras, one node manager computer for each rack location, andone port on the Local Area Network for each of the Video ActivityProcessors and Node Manager processors. The Arbitrator Processor sharesthe master computer of the CCTV system, and one copy of the Set RulesGUI resides on the GUI workstation in each of the three CCTV consoles.

In accordance with space limitations for the new system, and ifpermitted by available space, the video activity processors can beconventional rack mounted processors. For these processors, the systemmay use Pentium™ class processors available from Intel Corporation, orother high performance board-mounted processors, each capable of servingat least eight video cameras, i.e., controlling the acquisition of videooutput from such cameras. As an example, a system including processorsfor serving some 197 cameras in using dual on-board processors mayrequire 26 processors, each if rack-mounted being 7 inches in height andrequiring some 182 inches of rack space (about three full racks) andmust include a monitor.

In a more densely configured installation, the video activity processorsmay instead be commercially available single-board computers (“SBCs”) asheretofore used in industrial applications, so that, for example, eightcomputers in one chassis can serve 32 cameras. Other suitable processorconfigurations and types, either using complex instruction set (CISC) orreduced instruction set (RISC) software, may be employed.

Interfacing PCS system 10 to CCTV subsystem 12 is carried out by asingle processor providing a computer interface with an otherwisepre-existing CCTV system, and SentryConnector is used to connectArbitrator 40 to port server of CCTV subsystem 12. Thus, referring toFIG. 2, connections are established between each of four CCTV garagecameras 14 and Video Worker 34 module, which is in turn connected toVideo Supervisor 32, which is itself then connected to a Node Manager36.

CCTV garage cameras 14 are merely typical of possibly many video camerasof security system CCTV subsystem 12. There may for example be, as inthe example given above, hundreds of such cameras. While the new systemis especially well-suited for use in large-scale CCTV systems, as thustypified by hundreds of video cameras, it can also be used withsmall-scale CCTV systems having far fewer video cameras but whereelectronic analysis and supervision for controlling camera videopresentation is to be carried out by PCS system 10.

Video signals representing the view of each of CCTV garage cameras 14(as well as other video cameras of the system) are provided also to CCTVsystem 12, and thus are shown connected to distributed CCTV switches 16,which are illustrated as being supplied with video from cameras otherthan those shown. It should be appreciated that video outputs from allof the video cameras are provided to both PCS system 10 and to CCTVsubsystem 12 simultaneously.

The term PCS system has been used arbitrarily in describing the presentinvention, but other designations may be employed. By using computers topre-screen the cameras, only views with some event of interest to theoperators will be selected to the call-up monitors.

System Operation

The computer interface between the two systems, i.e. PCS system 10 andCCTV subsystem 12, functions in the following manner, with reference toFIG. 2: PCS system 10 requests a camera call up to one of the inputs toquad splitter 26 shown below GUI workstation 24. (The interface arrowpointing down)

Image analysis by PCS system 10 does not depend on the CCTV switchingsystem to be able to pre-screen the cameras, as the camera video goes toboth systems independently. The CCTV switching system does not depend onPCS system 10 to present video to the four quad monitors (16 views)depicted at the bottom of operator console 20.

Because CCTV subsystem 12, even without PCS system 10, can functionconventionally, when CCTV subsystem 12 is configured and tested fornormal operation, the interface between Arbitrator 40 and the GSIS portserver can be activated to test the operation of PCS system 10. With theCCTV switching system operational, and PCS system 10 operational, theautomatic video call-ups for the video cameras, such as those used forgarage surveillance, cause camera views to be displayed on the quadmonitor shown with a video input to GUI workstation 24.

PCS system 10 provides video image analysis to decrease staffingrequirements and (through reduced boredom) to increase the security ofpremises, such as garages, in which the new system is installed. PCSsystem 10 is software-based, with capability for image analysis in orderto allow persons to be distinguished from vehicles. With knowledge inthe system about where each camera is located, and what event the camerais viewing, the call-ups are based on a set of priority rules. Forexample, these rules may establish operation as follows for a securitysystem of the present invention when installed in a garage complex:

Each camera is assigned a location identifier to allow selection ofcameras to a particular console based on the garage it is in.

Each camera is assigned to a logical type group such as quiet aisle,entry aisle, or elevator lobby.

Event priorities are assigned to each logical group such as thesesituations:

Two or more persons in view converging from different start points.

One or more persons in view moving faster than normal.

Two or more persons in view, not converging.

One person walking alone.

Using a combination of location identifier and logical groups, thecamera call-ups at each console can be customized to control operatorloading. Garages may be assigned to individual consoles during daylighthours but during night hours all garages can be assigned to a singleconsole. Vehicles such as cars might normally be ignored during somehours of operation, but during a shift which is especially boringbecause of lack of video monitor activity, vehicles can be added to thepriority list to increase the frequency of monitor call-ups.

Set Rules GUI 38 can be included in each operator console 20 to allowsetting the rules for camera call-up. Preferably, access to Set RulesGUI 38 will be subject to password authorization.

Additional call-up events can be provided for PCS system 10 and providedas upgrades. When information is available from image analysis, othermore involved events may be available including situations such as:

A person has fallen down.

A person is walking erratically, such as may occur if “casing” cars orlost.

A person is taking too long to enter a car, which may represent break-ineffort.

A car is moving faster than a preset percentage (e.g., 95%) of othercars in the same camera view during a recent time interval.

Each operator console 20 preferably will have a call-up monitor withfour cameras displayed. A small “thumbnail” version of the four cameraview is displayed on GUI workstation 24. Camera call-ups are automatic.Each camera view selected remains on the console for a dwell time periodthat is user selected and entered in the rules. If an operator desiresto continue observing a specific camera view, a click on the quadrant ofthe thumbnail image on GUI workstation 24 will cause the selected camerato be switched to another larger monitor. For example, an operator canselect the view of two running persons for display on the large monitor.

In view of the foregoing description of the present invention andpractical embodiments it will be seen that the several objects of theinvention are achieved and other advantages are attained. Theembodiments and examples were chosen and described in order to bestexplain the principles of the invention and its practical application tothereby enable others skilled in the art to best utilize the inventionin various embodiments and with various modifications as are suited tothe particular use contemplated.

As various modifications could be made in the constructions and methodsherein described and illustrated without departing from the scope of theinvention, it is intended that all matter contained in the foregoingdescription or shown in the accompanying drawings shall be interpretedas illustrative rather than limiting.

The breadth and scope of the present invention should not be limited byany of the above-described exemplary embodiments, but should be definedonly in accordance with claims of the application and their equivalents.

1. A computer system for automated screening of a plurality of videocameras, said computer system being in communication with the pluralityof video cameras, the computer system comprising: a memory; and aplurality of real-time image analysis components, wherein video imagedata from said video cameras is analyzed by said real-time imageanalysis components and said video image data is then selectivelypresented to an operator for monitoring, at least a portion of at leastone of the plurality of real-time image analysis components is stored inat least a portion of the memory, the plurality of real-time imageanalysis components including: at least one video worker, said videoworker capturing video image data from said video cameras; at least oneanalysis worker, said analysis worker receiving said video image datafrom said video worker and performing image analysis on said video imagedata to provide data about activities in the video image data; at leastone video supervisor, said video supervisor controlling said receptionof video image data of said analysis worker from said video worker; atleast one node manager, said node manager receiving said data aboutactivities from said video worker; a set rules graphical user interface(GUI), said GUI providing an interface to said computer system forrequired configuration data for said computer system; and a systemarbitrator, said system arbitrator receiving said data about activitiesfrom said node manager and in communication with said set rules GUI;wherein said arbitrator selectively sends said video image data to anoperator for further use.
 2. The computer system according to claim 1,wherein said analysis worker includes a library of functions, whereinsaid functions manipulate said video image data.
 3. The computer systemaccording to claim 2 wherein said functions further comprise: colorspace calculations, said color space calculations comprising: a colordegree parameter which measures how far a color is from gray scale; anda color direction parameter which measures color based on atwo-dimensional color analysis.
 4. The computer system according toclaim 1, said computer system further comprising: a pan-tilt-zoomcontrol for movement control of the plurality of video cameras by saidoperator.
 5. The computer system according to claim 1, said computersystem further comprising: an operator console, said operator consolecomprising a plurality of video monitors; a central switch whichreceives input from said video cameras; and at least one quad splitter,said quad splitter receiving said video image data from said centralswitch and sending multiple video scenes to one of said video monitors.6. The computer system according to claim 1, wherein the computer systemis a security system, the video cameras are security cameras, andwherein said arbitrator selectively sends said video image data to anoperator for security monitoring purposes.
 7. A computer system forautomated screening of a plurality of video cameras, said computersystem being in communication with the plurality of video cameras, thecomputer system comprising: a memory; and a plurality of real-time imageanalysis components, wherein video image data from said video cameras isanalyzed by said real-time image analysis components and said videoimage data is then selectively presented to an operator for monitoring,at least a portion of at least one of the plurality of real-time imageanalysis components is stored in at least a portion of the memory, theplurality of real-time image analysis components including: at least onevideo worker, said video worker capturing video image data from saidvideo cameras; at least one analysis worker, said analysis workerreceiving said video image data from said video worker and performingimage analysis on said video image data to provide data about activitiesin the video image data; at least one video supervisor, said videosupervisor controlling said reception of video image data of saidanalysis worker from said video worker; at least one node manager, saidnode manager receiving said data about activities from said videoworker; a set rules graphical user interface (GUI), said GUI providingan interface to said computer system for required configuration data forsaid computer system; and a system arbitrator, said system arbitratorreceiving said data about activities from said node manager and incommunication with said set rules GUI; wherein said arbitratorselectively sends said video image data to an operator for furtherpurposes; wherein said analysis worker includes: (a) a provision forperforming at least one pass through such a frame of said video imagedata; and (b) a provision for generating a terrain map from said passthrough said frame of said video image data, said terrain map includinga plurality of parameters that indicate the content of said video imagedata; said terrain map containing in said plurality of parameterscharacteristic information regarding the content of the video imagedata, the characteristic information being based on each of kernels ofpixels in an input buffer, the characteristic information including atleast a number of bytes of data describing the relationship of each of aplurality of pixels in a larger kernel surrounding the first-saidkernel.
 8. The computer system according to claim 7, wherein saidfurther purposes are security monitoring purposes.
 9. A computer systemfor automated screening of a plurality of video cameras, said computersystem being in communication with the plurality of video cameras, thecomputer system comprising: a memory; and a plurality of real-time imageanalysis components, wherein video image data from said video cameras isanalyzed by said real-time image analysis components and said videoimage data is then selectively presented to an operator for monitoring,at least a portion of at least one of the plurality of real-time imageanalysis components is stored in at least a portion of the memory, theplurality of real-time image analysis components including: at least onevideo worker, said video worker capturing video image data from saidvideo cameras; at least one analysis worker, said analysis workerreceiving said video image data from said video worker and performingimage analysis on said video image data to provide data about activitiesin the video image data; at least one video supervisor, said videosupervisor controlling said reception of video image data of saidanalysis worker from said video worker; at least one node manager, saidnode manager receiving said data about activities from said videoworker; a set rules graphical user interface (GUI), said GUI providingan interface to said computer system for required configuration data forsaid computer system; and a system arbitrator, said system arbitratorreceiving said data about activities from said node manager and incommunication with said set rules GUI; wherein said arbitratorselectively sends said video image data to an operator for furtherpurposes; wherein said analysis worker includes: (a) a provision forperforming at least one pass through such a frame of said video imagedata; and (b) a provision for generating a terrain map from said passthrough said frame of said video image data, said terrain map includinga plurality of parameters that indicate the content of said video imagedata, said parameters including: an average altitude parameter; a degreeof slope parameter; a direction of slope parameter; a horizontalsmoothness parameter; a vertical smoothness parameter; and a jaggynessparameter.
 10. The computer system according to claim 9, wherein saidparameters are generated by color space calculations, said color spacecalculations comprising a color degree parameter which measures how fara color is from gray scale, and a color direction parameter whichmeasures color based on a two-dimensional color analysis.
 11. Thecomputer system according to claim 9, wherein said further purposes aresecurity monitoring purposes.